The present invention relates to a digital camera system having integrated accelerometers for determining static and dynamic accelerations of said digital camera system. Data relating to the determined static and dynamic accelerations are stored with recorded image data for further processing, such as for correcting image data for roll, pitch and vibrations. Data may also be used on-the-fly for smear suppression caused by vibrations.
When using rectangular film formats like the 35 mm format, images are recorded on film with a xe2x80x9clandscapexe2x80x9d (horizontal) orientation in respect to the common way of holding a camera. When the photographer wishes to capture a portrait he will tilt the camera 90 degrees and thus acquire an image with a xe2x80x9cportraitxe2x80x9d (vertical) orientation. Later when the developed images are viewed, the viewing person will manually orient them correctly. Since the images are on paper, it is relatively easy to reorient some of them.
In digital photography the landscape orientation is the default setting for most cameras. When the captured images are viewed on a display, they will appear with a landscape orientation with no respect to whether the images were actually captured with the camera held in a portrait or landscape orientation. The images then have to be manually inspected and later possibly rotated to reflect their original orientation. Some digital camera manufacturers are now beginning to include a sensor unit, which detects whether the camera is placed in landscape or portrait position when an image is captured.
In U.S. Pat. No. 5,900,909 an orientation detector which consists of two mercury tilt switches is described. The two mercury switches make it possible to determine whether the user is holding the camera in the normal landscape orientation or in a portrait orientation. There are two portrait orientations: One is the result of a clockwise rotation whereas the other is the result of a counter clockwise rotation. The use of mercury switches has some distinct disadvantages in that mercury can cause great damage when it interacts with the human body, and for that reason it is quite unpopular in many products. Mercury switches usually consume a lot of space in comparison with monolithic IC""s. This is due to their very mechanic structure, which makes miniaturisation difficult. In a digital camera it is crucial to minimise the size and weight, so in respect to this, the use of mercury and other primarily mechanically based switches, is not the optimum choice. A mercury switch based solution in a digital camera is limited to detecting a few rough orientations, i.e. landscape and portrait. The robustness and ease of use of the mercury switch are its primary advantages today.
The main limitation regarding micro-mechanical accelerometers fabricated in e.g. silicon is related to their ability to absorb shock without being damaged.
Taking pictures with long shutter times and maybe even a high degree of zoom makes the image capture process very sensitive to vibrations, which will result in blurred images. At short shutter times the image is less likely to be affected by vibrations since most vibrations, which will affect a camera, have an upper frequency limit, due to mechanical damping from the surroundings. Especially handheld photography easily results in blurred images when longer shutter speeds are used. One solution to the described limitations is to be able to compensate for most vibrations. Vibrations can be compensated optically by means of a lens module, which is capable of moving the projected image around in the image focus plane. This requires a special and expensive lens.
When vibrations cannot be compensated, another way of helping the photographer to acquire the optimum images is to inform him about any possibility of blurring, which may have occurred in a captured image. With feedback from the camera regarding the degree of shaking during the exposure time, it is possible for the photographer to decide whether he wants to capture another image of the same scene.
In U.S. Pat. No. 4,448,510, a camera shake detection apparatus is described. It includes an accelerometer, which is connected to a control circuit, which activates an alarm, when the acceleration exceeds a certain predefined threshold level. The threshold level can be influenced by the exposure timexe2x80x94a long exposure time results in a low threshold level and vice versa for a short exposure time. The output from the accelerometer may also be forced through an integrator before comparing the output to a threshold level to account for the fact that blurring is more probable to occur if a large number of high accelerations are detected. None of the described implementations are able to determine if the camera after a short period of vibrations returns to its initial position or the position where the majority of the exposure time has been spent. In such a case the suggested implementations would generate a xe2x80x9cblurxe2x80x9d alarm, even though the image could be sharp.
In some applications, especially the more technically oriented, it can be an advantage to have knowledge about how the camera is physically oriented in space. In a set-up with a digital camera connected to a GPS receiver, knowledge about the roll and pitch of a camera can be used to automatically pin point the scene being photographed. This can be used in aerial photography and other related technical applications. In other set-ups, feedback to the photographer about the exact roll and pitch can be useful for him to correct his orientation of the camera. Another use of the roll information is to automatically correct for small degrees of slant in the sideways direction. In most common photographic situations it is not desirable to have an automatic correction of a slight slant, as the photographer often wants full control of the image orientation. A feature like automatic slant correction should be user configurable in the sense that it can be turned off and on.
JP 58-222382 discloses an apparatus that automatically corrects inclination of scanned originals by changing the address where the image data is written to reflect the original with no inclination. Inclination is measured by using feedback from a couple of timing marks, which are connected to the slant of the original. Measuring the inclination through the use of timing marks is not useful in digital still photography. General image rotation in software is carried out by moving the original image data to a new position in another image file/buffer.
The present invention may be implemented in a digital still camera or a digital still camera back and supply a total solution which is very compact, consumes little power, and is applicable in a variety of digital still camera applications. The use of a single detector unit for a variety/plurality of functions decreases the physical size, lowers the power consumption, and keeps the prize down. The use of a micro-mechanical accelerometer as opposed to a mercury switch has the distinct advantage that it does not contain mercury.
The micro-mechanical accelerometer has several advantages over the mercury switch and the pendulum based orientation detector. Some of these advantages are:
it can easily be miniaturised,
it is a measurement device with a high degree of accuracy which can be configured dynamically for a variety of applications through the use of different processing which can be integrated in a digital processing unit or analogue electronics,
it may be applied to measure both static and dynamic acceleration at the same time. In comparison, the mercury switch and the pendulum are both optimised for measuring static orientation.
With the integration of more than one measurement axis in a silicon-based chip it becomes possible to measure both dynamic and static acceleration in several directions at the same time. The static acceleration is basically obtained by low-pass filtering the raw outputs from the accelerometer(s). More sophisticated filtering can be applied to handle specific requirements. With static acceleration from at least two axesxe2x80x94which are perpendicular to each otherxe2x80x94it is possible to obtain the precise degree of both roll and pitch for a digital still camera. This may be used in technical applications for automatic or manual correction of slant in both sideways and forwards directions. Mercury switches or pendulums are limited to a more rough evaluation of the orientation of the camera (basically limited to two positions).
A subset of the before-mentioned static acceleration measurement feature is the possibility to automatically determine when an image should be displayed with portrait or landscape orientation. The high precision of the roll and pitch information makes it possible to determine the correct orientation under the most difficult conditions where a slight mechanical tolerance for a mercury switch or pendulum based solution easily would result in an unexpected determination of orientation.
The mercury switch and pendulum switch based solutions lack the possibility to be dynamically configured to each users need, as their functionality is fixed mechanically when they leave the factory. An example of this could be a user who wishes that his camera should display images with a landscape orientation until he tilts the camera 75 degrees, whereas the normal configuration would be to display an image with a portrait orientation when the camera is tilted more than 45 degrees.
The measurements of dynamic acceleration (vibration) during the time of exposure may be used in a variety of ways to reduce the possibility of the photographer taking a blurred image. The use of active compensation for camera movements can be used to extend the previous working range for photography in terms of longer exposure time, more zoom, and the ability to capture images in vibration dominated surroundings, i.e. helicopters.
With a traditional film camera it is necessary to have an expensive lens which corrects the induced vibrations by changing the optical path of incident light. When the vibrations are compensated either by plain image processing with input from the recorded movements, or by active compensation through movement of charges in the image sensor, or by physically moving the image sensor itself, all the outlined compensation solutions described in detail below, enable the use of any type of lens, and are still able to reduce blur. The addition of a little extra image processing to compensate for vibrations through post-processing, or the use of charge movement in the sensor, does not increase the manufacturing cost, as opposed to a solution which changes the optical path.
When using accelerometers, generation of a xe2x80x9cblurxe2x80x9d warning is much more fail safe than earlier solutions which were not able to determine if the camera after a short period of vibrations would return to its initial position or the position where the majority of the exposure time had been spent. In such a case the earlier implementations would generate a xe2x80x9cblurxe2x80x9d alarm, even though the image could be sharp.
The present invention is therefore directed to a digital still camera which substantially overcomes one or more of the limitations and disadvantages of the related art. More particularly, the present invention is directed to a digital still camera with a sensor unit for determining static and dynamic accelerations, and methods thereof which substantially overcomes one or more of the limitations and disadvantages of the related art,
It is an object of the present invention to provide a sensor unit to digital cameras which is very compact, consumes little power, and is applicable in a variety of digital camera applications.
It is a further object of the present invention to provide a sensor unit to digital cameras capable of providing the following features:
Low-pass filtering the accelerometer outputs enables exact measurement of roll and pitch which can be used in technical applications for automatic or manual correction of slant in both sideways and forwards directions. The roll and pitch information is also useful in applications where knowledge of the camera shooting direction is needed, i.e. aerial photography.
A subset of the before mentioned feature is the possibility to automatically determine when an image should be displayed with portrait or landscape orientation.
A processing unit evaluates the raw accelerometer outputs during the time of exposure. The processing unit evaluates whether or not the measured vibrations may result in an image, which appears to be blurred. The photographer may receive a warning in case the processing unit finds that blur is highly likely to occur in the captured image.
The raw accelerometer outputs can also be used to keep track of the movements of the camera with respect to the field of gravity. When the image is processed afterwards it is possible to correct the image for blur by using the record of camera movements during the exposure time. During the exposure time, the camera movements can be actively compensated by moving charges (pixel information) in the image sensor in a direction to follow the movements of the projected image in the image plane. The movement of charges in the image sensor can be combined or replaced with mechanical actuators to physically move the image sensor.
In some cases a little blur may be advantageous to reduce the amount of Moirxc3xa9 image defects which may be introduced when an image is extremely sharp. Using the knowledge about the camera movements during the time of exposure it is possible for the image processor to generate an image with less tendency to show Moirxc3xa9 without the full reduction of sharpness.
In a first aspect, the present invention relates to a sensor unit to a digital camera, said sensor unit includes a detector which determines static and dynamic accelerations. The detector includes, a first sensor which senses acceleration in a first direction, and provides a first output signal in response to acceleration in the first direction; and a second sensor which senses acceleration in a second direction and provides a second output signal in response to acceleration in the second direction, the second direction being different from the first direction. The sensor unit also includes a processor which processes the first and second output signals. The processor includes a first filter which low-pass filters the first and second output signals so as to obtain information relating to static accelerations, and a second filter which band-pass filters the first and second output signals so as to obtain information relating to dynamic accelerations.
The first and second directions may be perpendicular to each other. The sensor unit may further include a third sensor which senses acceleration in a third direction and provides a third output signal in response to acceleration in the third direction, the third output signal being provided to the processor so as to obtain information relating to static and dynamic accelerations. The third direction may be perpendicular to the first and second directions.
The sensor unit may further include an alarm, which may generate an alarm signal in response to at least one of the output signals from the sensor. The alarm signal may be generated when at least one of the output signals exceeds a predetermined level which may relate to the fact that an image starts to get blurred or relate to a certain amount of exposure time. The alarm signal may be constituted by a sound signal, a flashing signal, an image file tag or any combination thereof.
At least one of the sensors may include a micro-mechanical deflection system. The first, second and third sensor may be integrated in a single micro-mechanical deflection system mounted in the camera house of the digital cameraxe2x80x94for example in a digital camera back.
At least one of the above and other objects may be realized by providing a method of determining static and dynamic accelerations in a digital camera, the method including:
providing a first sensor sensitive to acceleration in a first direction, said first sensor means being adapted to provide a first output signal in response to acceleration in the first direction,
providing a second sensor sensitive to acceleration in a second direction, said second sensor being adapted to provide a second output signal in response to acceleration in the second direction, the second direction being different from the first direction,
low-pass filtering the first and second output signals so as to obtain information relating to static accelerations, and
band-pass filtering the first and second output signals so as to obtain information relating to dynamic accelerations.
The method may further include providing a third sensor sensitive to acceleration in a third direction. The third sensor provides a third output signal in response to acceleration in the third direction, the third output signal being provided to the processor so as to obtain information relating to static and dynamic accelerations.
The first, second and third directions may be essentially perpendicular. The method according to the second aspect may further include generating an alarm signal as mentioned in relation to the first aspect of the present invention.
At least one of the above and other objects may be realized by providing a digital camera including
an image recording device, the image recording device comprising a plurality of light sensitive elements,
a first translator which translates the image recording device in a first direction in response to a first input signal,
a sensor unit according as set forth above, wherein the band-pass filtered first output signal from the first sensor is provided as the first input signal to the first translating so as to compensate for determined dynamic accelerations in the first direction.
The digital camera may further include
a second translator which translates the image recording device in a second direction in response to a second input signal, the second direction being different from the first direction,
a sensor unit as set forth above, where the band-pass filtered second output signal from the second sensor is provided as the second input signal to the second translator so as to compensate for determined dynamic accelerations in the second direction.
The first and second directions may be essentially perpendicular. The first and second translators may translate the image recording device in a plane substantially parallel to a plane defined by the plurality of light sensitive elements. The first and second translators may comprise micro-mechanical actuators.
At least one of the above and other objects may be realized by providing a method of processing image data, the method including:
providing image data, the image data being stored in a memory,
providing data or information relating to static accelerations as described above, providing data or information being recorded and stored with the image data, and
correcting the image data in accordance with the data or information relating to static accelerations so as to correct the image data and reduce the influence of roll and pitch.
Alternatively, the roll and pitch information may be used to determine whether the optimum way of displaying the image is with a portrait or landscape orientation.
At least one of the above and other objects may be realized by providing a method of correcting image data during recording of an image of an object, the method including:
recording image data of the object by projecting the object onto an array of light sensitive elements, recorded image data being generated as electrical charges in the array of light sensitive elements,
providing information relating to time dependent movements of the array of light sensitive elements relative to the object, and
correcting the recorded image data in accordance with the provided information relating to movements of the array of light sensitive elements relative to the object by moving charges (pixels) in the array of light sensitive elements so as to correct for relative movements between the array of light sensitive elements and the image of the object.
At least one of the above and other objects may be realized by providing a method of displaying a recorded image with a predetermined orientation, the method including:
providing information relating to the degree of roll of the recorded image, the information being provided by first and second sensor means sensitive to accelerations in a first and a second direction, respectively, the second direction being different from the first direction, and
using the provided information to determine the orientation by which the recorded image is to be displayed and/or stored.
The orientation by which the recorded image is to be displayed and/or stored may comprise portrait and landscape orientations. The user may determine at which predetermined acceleration levels the recorded image toggles between portrait and landscape orientation. The predetermined acceleration levels may correspond to a predetermined degree of roll of the recorded image.
At least one of the above and other objects may be realized by providing a method of correcting image data during recording of an image of an object, the method including:
recording image data of the object by projecting an image of the object onto an array of light sensitive elements,
providing information relating to time dependent movements of the array of light sensitive elements relative to the image of the object, and
correcting the recorded image data in accordance with the provided information relating to movements of the array of light sensitive elements relative to the image of the object by counter moving the array of light sensitive elements so as to compensate for the time dependent movements.
At least one of the above and other objects may be realized by providing a method of reducing Moirxc3xa9 image defects without full reduction in sharpness, the method including:
providing an array of light sensitive elements,
recording an image of an object using the array of light sensitive elements, the image being affected by movements of the array of light sensitive elements relative to the object so that the recorded image appears to be blurred and without Moirxc3xa9 defects,
providing information relating to time dependent movements of the array of light sensitive elements relative to the object during the time of exposure, and
using the provided information as an input to an image processing algorithm so as to reduce Moirxc3xa9 image defects in the recorded image and thereby obtain a modified image with increased sharpness.
At least one of the above and other objects may be realized by providing a computer program including code adapted to perform the method according to the any of the above methods when the program is run in a computer. The computer program may be embodied on a computer-readable medium.